@MatthewPyle-NOAA @dkokron When running UFS regression tests with this code, the control_restart_p8_intel test is failing with a segmentation fault. If you have access to Hera, you can see the logs here: /scratch1/BMC/gmtb/Grant.Firl/stmp2/Grant.Firl/FV3_RT/rt_1304460/control_restart_p8_intel

The err log shows the first non-libarary error as:
0x0000000002232a28 fv_io_mod_mp_fv_io_read_restart_() /scratch1/BMC/gmtb/Grant.Firl/ufs-weather-model-grantfirl/FV3/atmos_cubed_sphere/tools/fv_io.F90:495

This seems to be related to the changes that were made in this PR. Could you please help to debug this?

I don't have access to Hera. Does this failure happen on Acorn too?
…
On Fri, Jan 3, 2025 at 9:46 AM Grant Firl @.> wrote: @MatthewPyle-NOAA https://github.com/MatthewPyle-NOAA @dkokron https://github.com/dkokron When running UFS regression tests with this code, the control_restart_p8_intel test is failing with a segmentation fault. If you have access to Hera, you can see the logs here: /scratch1/BMC/gmtb/Grant.Firl/stmp2/Grant.Firl/FV3_RT/rt_1304460/control_restart_p8_intel The err log shows the first non-libarary error as: 0x0000000002232a28 fv_io_mod_mp_fv_io_read_restart_() /scratch1/BMC/gmtb/Grant.Firl/ufs-weather-model-grantfirl/FV3/atmos_cubed_sphere/tools/fv_io.F90:495 This seems to be related to the changes that were made in this PR. Could you please help to debug this? — Reply to this email directly, view it on GitHub <#896 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACODV2DOTNIIHTKZXNEHNNL2I2WGXAVCNFSM6AAAAABTCZ24DSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRZGQ2DENRTGI . You are receiving this because you were mentioned.Message ID: @.>

So far, it's been noted on Hera and Hercules. I'm guessing it would also show up on Acorn, but I don't have access.

@BrianCurtis-NOAA normally runs the UFS regression suite on acorn. Is he scheduled to do the same for this release?

@BrianCurtis-NOAA normally runs the UFS regression suite on acorn. Is he scheduled to do the same for this release?

This PR isn't necessarily associated with a release. It's just bringing changes from the RRFSv1 release branch into the main develop branch. Perhaps we can ask @BrianCurtis-NOAA to run the failing test only on Acorn for you to see if it fails there and you can have access to the run directory with logs?

I've run the UFS regression suite before. I was just wondering if Brian had done this already on acorn. Will the following clone get me to a state where I can reproduce the failure?

git clone --recursive [email protected]:ufs-community/ufs-weather-model.git CloneUFSForRegressionTesting

I've run the UFS regression suite before. I was just wondering if Brian had done this already on acorn. Will the following clone get me to a state where I can reproduce the failure?

git clone --recursive [email protected]:ufs-community/ufs-weather-model.git CloneUFSForRegressionTesting

I don't know if they had gotten to Acorn yet or not, but they stopped testing once this issue was noticed.

I'd use:
git clone --branch rrfsv1-to-ufs/dev6 --recursive https://github.com/grantfirl/ufs-weather-model.git
since this is the source branch for the PR. I'm sure that there are other ways.

Runs fine on acorn.
acorn:/lfs/h1/hpc/support/daniel.kokron/FV3_RT/rt_1846251/control_restart_p8_intel

Please try disabling the ENABLE_PARALLELRESTART feature in your Hera build.

line 39 of FV3/atmos_cubed_sphere/CMakeLists.txt
from
option(ENABLE_PARALLELRESTART "Enable collective parallel reads -DENABLE_PARALLELRESTART" ON)
to
option(ENABLE_PARALLELRESTART "Enable collective parallel reads -DENABLE_PARALLELRESTART" OFF)

Runs fine on acorn. acorn:/lfs/h1/hpc/support/daniel.kokron/FV3_RT/rt_1846251/control_restart_p8_intel

Please try disabling the ENABLE_PARALLELRESTART feature in your Hera build.

line 39 of FV3/atmos_cubed_sphere/CMakeLists.txt from option(ENABLE_PARALLELRESTART "Enable collective parallel reads -DENABLE_PARALLELRESTART" ON) to option(ENABLE_PARALLELRESTART "Enable collective parallel reads -DENABLE_PARALLELRESTART" OFF)

Interesting, thanks. I'll try your suggestion on Hera and report back on this PR.

@dkokron The offending test on Hera passes with -DENABLE_PARALLELRESTART" OFF.

@bensonr
Our agreement with the FMS team was that they would take ownership after they accepted the contributions that enable this feature. How do you want to proceed?